The latent words language model

Statistical language models have found many applications in information retrieval since their introduction almost three decades ago. Currently the most popular models are n-gram models, which are known to suffer from serious sparseness issues, which is a result of the large vocabulary size |V | of any given corpus and of the exponential nature of n-grams, where potentially |V | n-grams can occur in a corpus. Even when many n-grams in fact never occur due to grammatical and semantic restrictions in natural language, we still observe and exponential growth in unique n-grams with increasing n. Smoothing methods combine (specific, but sparse and potentially unreliable) higher order ngrams with (less specific but more reliable) lower order n-grams. (Goodman, 2001) found that interpolated Kneser-Ney smoothing (IKN) performed best in a comparison of different smoothing methods in terms of the perplexity on a previously unseen corpus. In this article we describe a novel language model that aims at solving this sparseness problem and in the process learns syntactic and semantic similar words, resulting in an improved language model in terms of perplexity reduction.